{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "lesbian-springer",
   "metadata": {},
   "source": [
    "## Seldon V2 Non Kubernetes Local Experiment Examples\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amber-inclusion",
   "metadata": {},
   "source": [
    "### Model Experiment\n",
    "\n",
    "We will use two SKlearn Iris classification models to illustrate experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dfc5a4fd",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: iris\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/mlserver/iris\"\r\n",
      "  requirements:\r\n",
      "  - sklearn\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/sklearn1.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8837f315",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: iris2\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/mlserver/iris\"\r\n",
      "  requirements:\r\n",
      "  - sklearn\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/sklearn2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "359b456e",
   "metadata": {},
   "source": [
    "Load both models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "elect-samba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/sklearn1.yaml\n",
    "!seldon model load -f ./models/sklearn2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aaa2b8f",
   "metadata": {},
   "source": [
    "Wait for both models to be ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "floral-moisture",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status iris -w ModelAvailable\n",
    "!seldon model status iris2 -w ModelAvailable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0a7cfc57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:iris_1::50]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris -i 50 \\\n",
    "  '{\"inputs\": [{\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "570c4769",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:iris2_1::50]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris2 -i 50 \\\n",
    "  '{\"inputs\": [{\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf1ac158",
   "metadata": {},
   "source": [
    "Create an experiment that modifies the iris model to add a second model splitting traffic 50/50 between the two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "10fe4447",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Experiment\r\n",
      "metadata:\r\n",
      "  name: experiment-sample\r\n",
      "spec:\r\n",
      "  default: iris\r\n",
      "  candidates:\r\n",
      "  - name: iris\r\n",
      "    weight: 50\r\n",
      "  - name: iris2\r\n",
      "    weight: 50\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./experiments/ab-default-model.yaml "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c9b215b",
   "metadata": {},
   "source": [
    "Start the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "differential-array",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment start -f ./experiments/ab-default-model.yaml "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "727e4fd4",
   "metadata": {},
   "source": [
    "Wait for the experiment to be ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "overhead-banks",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"experimentName\": \"experiment-sample\",\r\n",
      "  \"active\": true,\r\n",
      "  \"candidatesReady\": true,\r\n",
      "  \"mirrorReady\": true,\r\n",
      "  \"statusDescription\": \"experiment active\",\r\n",
      "  \"kubernetesMeta\": {}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon experiment status experiment-sample -w | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe293b0",
   "metadata": {},
   "source": [
    "Run a set of calls and record which route the traffic took. There should be roughly a 50/50 split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "finite-significance",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:iris2_1::27 :iris_1::23]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris -i 50 \\\n",
    "  '{\"inputs\": [{\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ac708c",
   "metadata": {},
   "source": [
    "Show sticky session header `x-seldon-route` that is returned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "85357805",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> POST /v2/models/iris/infer HTTP/1.1\r\n",
      "> Host: localhost:9000\r\n",
      "> Content-Type:[application/json]\r\n",
      "> Seldon-Model:[iris]\r\n",
      "\r\n",
      "< X-Seldon-Route:[:iris_1:]\r\n",
      "< Ce-Id:[463e96ad-645f-4442-8890-4c340b58820b]\r\n",
      "< Traceparent:[00-fe9e87fcbe4be98ed82fb76166e15ceb-d35e7ac96bd8b718-01]\r\n",
      "< X-Envoy-Upstream-Service-Time:[3]\r\n",
      "< Ce-Specversion:[0.3]\r\n",
      "< Date:[Thu, 29 Jun 2023 14:03:03 GMT]\r\n",
      "< Ce-Source:[io.seldon.serving.deployment.mlserver]\r\n",
      "< Content-Type:[application/json]\r\n",
      "< Server:[envoy]\r\n",
      "< X-Request-Id:[cieou5ofh5ss73fbjdu0]\r\n",
      "< Ce-Endpoint:[iris_1]\r\n",
      "< Ce-Modelid:[iris_1]\r\n",
      "< Ce-Type:[io.seldon.serving.inference.response]\r\n",
      "< Content-Length:[213]\r\n",
      "< Ce-Inferenceservicename:[mlserver]\r\n",
      "< Ce-Requestid:[463e96ad-645f-4442-8890-4c340b58820b]\r\n",
      "\r\n",
      "{\r\n",
      "\t\"model_name\": \"iris_1\",\r\n",
      "\t\"model_version\": \"1\",\r\n",
      "\t\"id\": \"463e96ad-645f-4442-8890-4c340b58820b\",\r\n",
      "\t\"parameters\": {},\r\n",
      "\t\"outputs\": [\r\n",
      "\t\t{\r\n",
      "\t\t\t\"name\": \"predict\",\r\n",
      "\t\t\t\"shape\": [\r\n",
      "\t\t\t\t1,\r\n",
      "\t\t\t\t1\r\n",
      "\t\t\t],\r\n",
      "\t\t\t\"datatype\": \"INT64\",\r\n",
      "\t\t\t\"parameters\": {\r\n",
      "\t\t\t\t\"content_type\": \"np\"\r\n",
      "\t\t\t},\r\n",
      "\t\t\t\"data\": [\r\n",
      "\t\t\t\t2\r\n",
      "\t\t\t]\r\n",
      "\t\t}\r\n",
      "\t]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris --show-headers \\\n",
    "  '{\"inputs\": [{\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ca1665",
   "metadata": {},
   "source": [
    "Use sticky session key passed by last infer request to ensure same route is taken each time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1ffb00b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:iris_1::50]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris -s -i 50 \\\n",
    "  '{\"inputs\": [{\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "81a10627",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:iris_1::50]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris --inference-mode grpc -s -i 50\\\n",
    "   '{\"model_name\":\"iris\",\"inputs\":[{\"name\":\"input\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[1,4]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12ad0016",
   "metadata": {},
   "source": [
    "Stop the experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "downtown-construction",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment stop experiment-sample"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da648e8",
   "metadata": {},
   "source": [
    "Unload both models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "077bbce8",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload iris\n",
    "!seldon model unload iris2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819d5b28",
   "metadata": {},
   "source": [
    "### Pipeline Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aee426cd",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: add10\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/add10\"\r\n",
      "  requirements:\r\n",
      "  - triton\r\n",
      "  - python\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/add10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3720658f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: mul10\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/mul10\"\r\n",
      "  requirements:\r\n",
      "  - triton\r\n",
      "  - python\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e1a8c098",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/add10.yaml\n",
    "!seldon model load -f ./models/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "46ac6ee1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status add10 -w ModelAvailable\n",
    "!seldon model status mul10 -w ModelAvailable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "44cae93b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: pipeline-mul10\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: mul10\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - mul10\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "64780610",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: pipeline-add10\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: add10\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - add10\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/add10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "18bbd7e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/add10.yaml\n",
    "!seldon pipeline load -f ./pipelines/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2e82b63b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"pipelineName\":\"pipeline-add10\", \"versions\":[{\"pipeline\":{\"name\":\"pipeline-add10\", \"uid\":\"cieov47l80lc739juklg\", \"version\":1, \"steps\":[{\"name\":\"add10\"}], \"output\":{\"steps\":[\"add10.outputs\"]}, \"kubernetesMeta\":{}}, \"state\":{\"pipelineVersion\":1, \"status\":\"PipelineReady\", \"reason\":\"created pipeline\", \"lastChangeTimestamp\":\"2023-06-29T14:05:04.460868091Z\", \"modelsReady\":true}}]}\n",
      "{\"pipelineName\":\"pipeline-mul10\", \"versions\":[{\"pipeline\":{\"name\":\"pipeline-mul10\", \"uid\":\"cieov47l80lc739jukm0\", \"version\":1, \"steps\":[{\"name\":\"mul10\"}], \"output\":{\"steps\":[\"mul10.outputs\"]}, \"kubernetesMeta\":{}}, \"state\":{\"pipelineVersion\":1, \"status\":\"PipelineReady\", \"reason\":\"created pipeline\", \"lastChangeTimestamp\":\"2023-06-29T14:05:04.631980330Z\", \"modelsReady\":true}}]}\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status pipeline-add10 -w PipelineReady \n",
    "!seldon pipeline status pipeline-mul10 -w PipelineReady "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "67c8cedd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M . "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a549a962",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          10,\r\n",
      "          20,\r\n",
      "          30,\r\n",
      "          40\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-mul10 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1153547d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Experiment\r\n",
      "metadata:\r\n",
      "  name: addmul10\r\n",
      "spec:\r\n",
      "  default: pipeline-add10\r\n",
      "  resourceType: pipeline\r\n",
      "  candidates:\r\n",
      "  - name: pipeline-add10\r\n",
      "    weight: 50\r\n",
      "  - name: pipeline-mul10\r\n",
      "    weight: 50\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./experiments/addmul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e3229c09",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment start -f ./experiments/addmul10.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1c4dbd59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"experimentName\": \"addmul10\",\r\n",
      "  \"active\": true,\r\n",
      "  \"candidatesReady\": true,\r\n",
      "  \"mirrorReady\": true,\r\n",
      "  \"statusDescription\": \"experiment active\",\r\n",
      "  \"kubernetesMeta\": {}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon experiment status addmul10 -w | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "20ba8a38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:add10_1::28 :mul10_1::22 :pipeline-add10.pipeline::28 :pipeline-mul10.pipeline::22]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -i 50 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b928d9b7",
   "metadata": {},
   "source": [
    "Use sticky session key passed by last infer request to ensure same route is taken each time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2f2a3a27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> /inference.GRPCInferenceService/ModelInfer HTTP/2\r\n",
      "> Host: localhost:9000\r\n",
      "> seldon-model:[pipeline-add10.pipeline]\r\n",
      "\r\n",
      "< x-envoy-expected-rq-timeout-ms:[60000]\r\n",
      "< x-request-id:[cieov8ofh5ss739277i0]\r\n",
      "< date:[Thu, 29 Jun 2023 14:05:23 GMT]\r\n",
      "< server:[envoy]\r\n",
      "< content-type:[application/grpc]\r\n",
      "< x-envoy-upstream-service-time:[6]\r\n",
      "< x-seldon-route:[:add10_1: :pipeline-add10.pipeline:]\r\n",
      "< x-forwarded-proto:[http]\r\n",
      "\r\n",
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[11, 12, 13, 14]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 --show-headers --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "082c6bb7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> /inference.GRPCInferenceService/ModelInfer HTTP/2\r\n",
      "> Host: localhost:9000\r\n",
      "> x-seldon-route:[:add10_1: :pipeline-add10.pipeline:]\r\n",
      "> seldon-model:[pipeline-add10.pipeline]\r\n",
      "\r\n",
      "< content-type:[application/grpc]\r\n",
      "< x-forwarded-proto:[http]\r\n",
      "< x-envoy-expected-rq-timeout-ms:[60000]\r\n",
      "< x-seldon-route:[:add10_1: :pipeline-add10.pipeline: :pipeline-add10.pipeline:]\r\n",
      "< x-request-id:[cieov90fh5ss739277ig]\r\n",
      "< x-envoy-upstream-service-time:[7]\r\n",
      "< date:[Thu, 29 Jun 2023 14:05:24 GMT]\r\n",
      "< server:[envoy]\r\n",
      "\r\n",
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[11, 12, 13, 14]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -s --show-headers --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d70f0acf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:add10_1::50 :pipeline-add10.pipeline::150]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -s -i 50 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "dedfbed5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: add20\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/triton/add20\"\r\n",
      "  requirements:\r\n",
      "  - triton\r\n",
      "  - python\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/add20.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "1436e721",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/add20.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "9d010fd2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model status add20 -w ModelAvailable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "09094919",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Experiment\r\n",
      "metadata:\r\n",
      "  name: add1020\r\n",
      "spec:\r\n",
      "  default: add10\r\n",
      "  candidates:\r\n",
      "  - name: add10\r\n",
      "    weight: 50\r\n",
      "  - name: add20\r\n",
      "    weight: 50\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./experiments/add1020.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "4547d936",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment start -f ./experiments/add1020.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "fba5dce6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"experimentName\": \"add1020\",\r\n",
      "  \"active\": true,\r\n",
      "  \"candidatesReady\": true,\r\n",
      "  \"mirrorReady\": true,\r\n",
      "  \"statusDescription\": \"experiment active\",\r\n",
      "  \"kubernetesMeta\": {}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon experiment status add1020 -w | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8f089362",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:add10_1::22 :add20_1::28]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer add10 -i 50  --inference-mode grpc \\\n",
    "  '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "66d7fa6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:add10_1::24 :add20_1::32 :mul10_1::44 :pipeline-add10.pipeline::56 :pipeline-mul10.pipeline::44]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -i 100 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "c0f22776",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> /inference.GRPCInferenceService/ModelInfer HTTP/2\r\n",
      "> Host: localhost:9000\r\n",
      "> seldon-model:[pipeline-add10.pipeline]\r\n",
      "\r\n",
      "< x-request-id:[cieovf0fh5ss739279u0]\r\n",
      "< x-envoy-upstream-service-time:[5]\r\n",
      "< x-seldon-route:[:add10_1: :pipeline-add10.pipeline:]\r\n",
      "< date:[Thu, 29 Jun 2023 14:05:48 GMT]\r\n",
      "< server:[envoy]\r\n",
      "< content-type:[application/grpc]\r\n",
      "< x-forwarded-proto:[http]\r\n",
      "< x-envoy-expected-rq-timeout-ms:[60000]\r\n",
      "\r\n",
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[11, 12, 13, 14]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 --show-headers --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "774e4373",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> /inference.GRPCInferenceService/ModelInfer HTTP/2\r\n",
      "> Host: localhost:9000\r\n",
      "> x-seldon-route:[:add10_1: :pipeline-add10.pipeline:]\r\n",
      "> seldon-model:[pipeline-add10.pipeline]\r\n",
      "\r\n",
      "< x-forwarded-proto:[http]\r\n",
      "< x-envoy-expected-rq-timeout-ms:[60000]\r\n",
      "< x-request-id:[cieovf8fh5ss739279ug]\r\n",
      "< x-envoy-upstream-service-time:[6]\r\n",
      "< date:[Thu, 29 Jun 2023 14:05:49 GMT]\r\n",
      "< server:[envoy]\r\n",
      "< content-type:[application/grpc]\r\n",
      "< x-seldon-route:[:add10_1: :pipeline-add10.pipeline: :add20_1: :pipeline-add10.pipeline:]\r\n",
      "\r\n",
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[21, 22, 23, 24]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -s --show-headers --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9f3f766e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment stop addmul10\n",
    "!seldon experiment stop add1020\n",
    "!seldon pipeline unload pipeline-add10\n",
    "!seldon pipeline unload pipeline-mul10\n",
    "!seldon model unload add10\n",
    "!seldon model unload add20\n",
    "!seldon model unload mul10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b16bbc",
   "metadata": {},
   "source": [
    "### Model Mirror Experiment\n",
    "\n",
    "We will use two SKlearn Iris classification models to illustrate a model with a mirror."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "20f19234",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: iris\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/mlserver/iris\"\r\n",
      "  requirements:\r\n",
      "  - sklearn\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/sklearn1.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "40ba5a74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: iris2\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/mlserver/iris\"\r\n",
      "  requirements:\r\n",
      "  - sklearn\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/sklearn2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "328fdaf6",
   "metadata": {},
   "source": [
    "Load both models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b4ec8d19",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/sklearn1.yaml\n",
    "!seldon model load -f ./models/sklearn2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed848e0b",
   "metadata": {},
   "source": [
    "Wait for both models to be ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "115699a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status iris -w ModelAvailable\n",
    "!seldon model status iris2 -w ModelAvailable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c58bb91",
   "metadata": {},
   "source": [
    "Create an experiment that modifies in which we mirror traffic to iris also to iris2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "632b4a58",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Experiment\r\n",
      "metadata:\r\n",
      "  name: sklearn-mirror\r\n",
      "spec:\r\n",
      "  default: iris\r\n",
      "  candidates:\r\n",
      "  - name: iris\r\n",
      "    weight: 100\r\n",
      "  mirror:\r\n",
      "    name: iris2\r\n",
      "    percent: 100\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./experiments/sklearn-mirror.yaml "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f83c88c",
   "metadata": {},
   "source": [
    "Start the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "2466a9b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment start -f ./experiments/sklearn-mirror.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e8e4e63",
   "metadata": {},
   "source": [
    "Wait for the experiment to be ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "0ac0054e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"experimentName\": \"sklearn-mirror\",\r\n",
      "  \"active\": true,\r\n",
      "  \"candidatesReady\": true,\r\n",
      "  \"mirrorReady\": true,\r\n",
      "  \"statusDescription\": \"experiment active\",\r\n",
      "  \"kubernetesMeta\": {}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon experiment status sklearn-mirror -w | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df95e80d",
   "metadata": {},
   "source": [
    "We get responses from iris but all requests would also have been mirrored to iris2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "81c6023d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: map[:iris_1::50]\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model infer iris -i 50 \\\n",
    "  '{\"inputs\": [{\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e445bb7",
   "metadata": {},
   "source": [
    "We can check the local prometheus port from the agent to validate requests went to iris2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "35a0f1f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon_model_infer_total{code=\"200\",method_type=\"rest\",model=\"iris\",model_internal=\"iris2_1\",server=\"mlserver\",server_replica=\"0\"} 50\r\n"
     ]
    }
   ],
   "source": [
    "!curl -s 0.0.0:9006/metrics | grep seldon_model_infer_total | grep iris2_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dee8fd1",
   "metadata": {},
   "source": [
    "Stop the experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "e777b0c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment stop sklearn-mirror"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af10e93",
   "metadata": {},
   "source": [
    "Unload both models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "44085e38",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload iris\n",
    "!seldon model unload iris2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "006bed5c",
   "metadata": {},
   "source": [
    "## Pipeline Mirror Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "01bf3aa8",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: add10\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/add10\"\r\n",
      "  requirements:\r\n",
      "  - triton\r\n",
      "  - python\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/add10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "be5c5426",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: mul10\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/mul10\"\r\n",
      "  requirements:\r\n",
      "  - triton\r\n",
      "  - python\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "54ac44c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/add10.yaml\n",
    "!seldon model load -f ./models/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "78debf8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status add10 -w ModelAvailable\n",
    "!seldon model status mul10 -w ModelAvailable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "c04346fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: pipeline-mul10\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: mul10\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - mul10\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "0aa43131",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: pipeline-add10\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: add10\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - add10\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/add10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "8bb044d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/add10.yaml\n",
    "!seldon pipeline load -f ./pipelines/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "cd5c869f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"pipelineName\":\"pipeline-add10\", \"versions\":[{\"pipeline\":{\"name\":\"pipeline-add10\", \"uid\":\"ciep072i8ufs73flaipg\", \"version\":1, \"steps\":[{\"name\":\"add10\"}], \"output\":{\"steps\":[\"add10.outputs\"]}, \"kubernetesMeta\":{}}, \"state\":{\"pipelineVersion\":1, \"status\":\"PipelineReady\", \"reason\":\"created pipeline\", \"lastChangeTimestamp\":\"2023-06-29T14:07:24.903503109Z\", \"modelsReady\":true}}]}\n",
      "{\"pipelineName\":\"pipeline-mul10\", \"versions\":[{\"pipeline\":{\"name\":\"pipeline-mul10\", \"uid\":\"ciep072i8ufs73flaiq0\", \"version\":1, \"steps\":[{\"name\":\"mul10\"}], \"output\":{\"steps\":[\"mul10.outputs\"]}, \"kubernetesMeta\":{}}, \"state\":{\"pipelineVersion\":1, \"status\":\"PipelineReady\", \"reason\":\"created pipeline\", \"lastChangeTimestamp\":\"2023-06-29T14:07:25.082642153Z\", \"modelsReady\":true}}]}\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status pipeline-add10 -w PipelineReady \n",
    "!seldon pipeline status pipeline-mul10 -w PipelineReady "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "9c2b666e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[11, 12, 13, 14]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "ef844ae7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[10, 20, 30, 40]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-mul10 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "3a9f756f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Experiment\r\n",
      "metadata:\r\n",
      "  name: addmul10-mirror\r\n",
      "spec:\r\n",
      "  default: pipeline-add10\r\n",
      "  resourceType: pipeline\r\n",
      "  candidates:\r\n",
      "  - name: pipeline-add10\r\n",
      "    weight: 100\r\n",
      "  mirror:\r\n",
      "    name: pipeline-mul10\r\n",
      "    percent: 100\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./experiments/addmul10-mirror.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "3777e310",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment start -f ./experiments/addmul10-mirror.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "63d087f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"experimentName\": \"addmul10-mirror\",\r\n",
      "  \"active\": true,\r\n",
      "  \"candidatesReady\": true,\r\n",
      "  \"mirrorReady\": true,\r\n",
      "  \"statusDescription\": \"experiment active\",\r\n",
      "  \"kubernetesMeta\": {}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon experiment status addmul10-mirror -w | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "319ef08f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[11, 12, 13, 14]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -i 1 --inference-mode grpc \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23eb87d0",
   "metadata": {},
   "source": [
    "Let's check that the mul10 model was called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "086359f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon_model_infer_total{code=\"OK\",method_type=\"grpc\",model=\"mul10\",model_internal=\"mul10_1\",server=\"triton\",server_replica=\"0\"} 2\r\n"
     ]
    }
   ],
   "source": [
    "!curl -s 0.0.0:9007/metrics | grep seldon_model_infer_total | grep mul10_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "25d03bc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon_model_infer_total{code=\"OK\",method_type=\"grpc\",model=\"add10\",model_internal=\"add10_1\",server=\"triton\",server_replica=\"0\"} 2\r\n"
     ]
    }
   ],
   "source": [
    "!curl -s 0.0.0:9007/metrics | grep seldon_model_infer_total | grep add10_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bacaccb",
   "metadata": {},
   "source": [
    "Let's do an http call and check agaib the two models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "94d784bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "\t\"model_name\": \"\",\r\n",
      "\t\"outputs\": [\r\n",
      "\t\t{\r\n",
      "\t\t\t\"data\": [\r\n",
      "\t\t\t\t11,\r\n",
      "\t\t\t\t12,\r\n",
      "\t\t\t\t13,\r\n",
      "\t\t\t\t14\r\n",
      "\t\t\t],\r\n",
      "\t\t\t\"name\": \"OUTPUT\",\r\n",
      "\t\t\t\"shape\": [\r\n",
      "\t\t\t\t4\r\n",
      "\t\t\t],\r\n",
      "\t\t\t\"datatype\": \"FP32\"\r\n",
      "\t\t}\r\n",
      "\t]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-add10 -i 1 \\\n",
    " '{\"model_name\":\"add10\",\"inputs\":[{\"name\":\"INPUT\",\"data\":[1,2,3,4],\"datatype\":\"FP32\",\"shape\":[4]}]}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "509efbcb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon_model_infer_total{code=\"OK\",method_type=\"grpc\",model=\"mul10\",model_internal=\"mul10_1\",server=\"triton\",server_replica=\"0\"} 3\r\n"
     ]
    }
   ],
   "source": [
    "!curl -s 0.0.0:9007/metrics | grep seldon_model_infer_total | grep mul10_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "d4d86a74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon_model_infer_total{code=\"OK\",method_type=\"grpc\",model=\"add10\",model_internal=\"add10_1\",server=\"triton\",server_replica=\"0\"} 3\r\n"
     ]
    }
   ],
   "source": [
    "!curl -s 0.0.0:9007/metrics | grep seldon_model_infer_total | grep add10_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "eb8be18f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon.default.model.mul10.inputs\tciep0bofh5ss73dpdiq0\t{\"inputs\":[{\"name\":\"INPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[1, 2, 3, 4]}}]}\r\n",
      "seldon.default.model.mul10.outputs\tciep0bofh5ss73dpdiq0\t{\"modelName\":\"mul10_1\", \"modelVersion\":\"1\", \"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[10, 20, 30, 40]}}]}\r\n",
      "seldon.default.pipeline.pipeline-mul10.inputs\tciep0bofh5ss73dpdiq0\t{\"inputs\":[{\"name\":\"INPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[1, 2, 3, 4]}}]}\r\n",
      "seldon.default.pipeline.pipeline-mul10.outputs\tciep0bofh5ss73dpdiq0\t{\"outputs\":[{\"name\":\"OUTPUT\", \"datatype\":\"FP32\", \"shape\":[\"4\"], \"contents\":{\"fp32Contents\":[10, 20, 30, 40]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline inspect pipeline-mul10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "c84bad04",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon experiment stop addmul10-mirror\n",
    "!seldon pipeline unload pipeline-add10\n",
    "!seldon pipeline unload pipeline-mul10\n",
    "!seldon model unload add10\n",
    "!seldon model unload mul10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc4a9b2e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
